5.2 Quality Assurance Application

The QA manager was recruited three months into the survey, after an initial three-week assessment of the field operations the QA manager identified broad objectives that were practical within the limitations of field operations in Cambodia:

• 3% of the information collected was to be QA assessed in the field,



• That QA would maintain a constant application and presence in the field,



• There would an even distribution between the survey teams, to include assessments in difficult and remote locations, and



• Assessments were to focus on mine / UXO positive reports, as these included the greatest amount of information, therefore potentially could contain the most errors of consequence.

The QA team commenced field assessments in December 2000, and continued until the completion of field operations in April 2002.

Documentation

Quality Control and Quality Assurance operating procedures are detailed in Annex 5, SOP 11 Level One Survey, SOP 12 Quality Assurance and Quality Control Processes, and SOP 15 Internal QA/QC Procedures.

In addition the records kept by the QA manager are archived with the survey database, this includes the checklists used, and log of villages visited, summaries of distribution, and chronologies etc.

Sample Size and Distribution

Over an 18-month period, a total of 465 villages were visited, and 486 village survey reports assessed. Some reports later became redundant as a result of the re survey activities. 421 QA visits are on record in the survey Database.

Every province and municipality in Cambodia was sampled, 157 communes in 83 districts were visited:

• 3.4% of reports generated were assessed by QA in the field,



• 5.1% of positive reports generated were assessed, and



• 1.4% of negative reports generated were assessed.

On average each survey detachment was assessed in the field 13.5 times. The geographic distribution of the field assessments is illustrated in the map at QA/QC Field Checks

QA was predisposed towards the reports that contained the most information i.e. mine / UXO contaminated villages. 77% (326 reports) of the QA effort went towards positive villages, 23% (99 reports) towards negative villages.

On 127 occasions it was possible to conduct complete check on reports that contained suspected areas, (mine, UXO, or Cluster Bomb areas.), with a socio-economic impact.

The chronology of field assessment conducted is at Chronology and Distribution of QA Checks

Unsatisfactory Village Survey Reports

QA assessments in the field, produced one of two results, reports were classified as being satisfactory or unsatisfactory. Satisfactory reports were those that were assessed and found to be correct in every aspect i.e. required no alteration or editing by QA.

Unsatisfactory reports were those where an assessment identified an error or omission and those for which additional enquiries were required.

Not every report classified by QA as unsatisfactory contained errors.

Not every error was caused by the surveyors; omissions by respondents produced the most serious errors.

Some reports were classified as unsatisfactory not because of errors in the report but because it was identified that there had been a failure in compliance with the procedures.

For example in Kampong Cham province the QA team during the conduct of an assessment identified an incorrect coordinate for the village leaders house, the report was classified as unsatisfactory and returned to the team leader for remedial action. The Team Leader upon revisiting the village identified that the coordinate was correct and that the village leader had two wives and two houses within the village.

The majority of unsatisfactory reports were corrected by QA during the course of the assessment, these reports remained classified as unsatisfactory even after the corrections were made. This was done in order to provide the surveyors with feedback and performance statements.

Integrity

100% Integrity, without exception every report assessed proved to be the product of survey activity, every inquiry made by the QA manager relating to the integrity of the reports proved that the information in the report was the result of a visit to the village by a survey detachment.

Compliance with Procedures

99.5% of the sampled reports were collected by surveyors using the prescribed operating procedures. It was possible during the QA process to verify that the survey was conducted in accordance with the operating procedures, specifically how many people were involved in the survey, how much time was taken, and what impression the surveyors left with the villagers. General enquiries resulted in responses that indicted that the surveyors were professional, polite, and courteous.

Only two reports contained the names of village representatives that had not been interviewed, investigation of these reports determined that:

• One respondent was reluctant to give her name and instead submitted her husbands' name,



• One respondent identified on the form was absent at the time of the survey, but was the owner of the land and was entered on the form as an information provider

Individual Error in Compasses

100% accurate equipment. The survey detachments employed compasses to provide information into the Survey reports, in total 57 compasses were used. No compass used in the survey had an individual error greater than 2o, the majority of the compasses used had 1o of error or less.

Coordinates

98% of the coordinates checked were satisfactory. Over the project period approximately 1,000 coordinates were reproduced by Quality Assurance. 22 coordinates were found to be incorrect, the major cause for the error was incorrect transcription.

I.e. E493668 became E493568.

Global Positioning Systems

Global Positioning Systems (GPS) were used to provide coordinates that were included in every report. The standard required by Quality Assurance was that the coordinates in the survey reports had to be easily reproduced to within 25 metres.

All GPS were tested by QA in order to determine the level of accuracy and precision. With one exception all GPS units produced accurate reproducible coordinates. The failed unit was being used by detachment 26, this unit produced coordinates that were within 100m but was removed from service in November 2001 when it was identified that as a consequence coordinates associated with a village sketch were not reproducible and Survey credibility was affected. The detachments work performance was investigated from May 18 (their previous assessment had been satisfactory) forward.

Negative villages were disregarded due to the minor nature of the error, twenty-three survey reports were investigated through the database, eight villages were queried because they contained Benchmark coordinates, and finally three villages were revisited by the Team Leaders in order to correct the benchmark coordinates.

Mapping Materials

Surveyors in the field represented area information on 1:50,000 American or Vietnamese maps, made in the 1960's or 1970. On occasion these maps contain errors and omissions relating to infrastructure that has come into place since the map was produced. No attempt was made to correct these omissions.

Village Sketches

97% of the Village Sketches assessed in the field were satisfactory.

2,983 Village Sketches were produced by the surveyors in villages that were mine / UXO area contaminated.127 (4.3%) of these sketches were quality assured in the field. In general the surveyors produce sketches that are accurate, very clear and easily understood.

Five village sketches were unsatisfactory because they contained omissions that had socio-economic consequences.

Early in the survey (December 2000) reports were received by the database that contained errors in the Village Sketches, the errors related to a failure by the surveyors to plot coordinates on the map and then transcribe the perspective (distance / bearing) onto the village sketch (map enlargement). This resulted in refresher training for the staff and the resurvey of 68 villages in Pailin municipality.

Village Survey Report Cover Check Sheet Q1 – Q4

Village information contained in the questionnaire, questions Q1 – Q4, include community identification, Mine / UXO information, Verification and Attached Documents listing. It is this segment of the report that contains the village landmark, and village point coordinates. These reports proved 98% correct during the QA process.

Of the 486 reports checked, one report was missing an attached document, (Spot UXO report), all other errors related to coordinates.

Questionnaire Check Q5 – Q9

Questions Q5 – Q9 detail / record the authors of the report and the Quality Control measures taken to produce the report, i.e. Checked by team leader and or Field Manager (1%). This section of the report was transcribed onto the QA checklist; some village survey reports were assessed prior to the finalisation of this part of the report. 100% checks of this information were conducted by Database.

Village Information Q10 – Q15

Village information contained in the questionnaire, questions Q10 – Q15, relate to Population, Poverty, Problems, History, Mine Action and Respondents Details. Of the 127 reports checked none required editing of the questions. These reports proved 100% correct during the QA process.

Suspected Mine (UXO) Area Information Q16 – Q17

Area information contained in the questionnaire, questions Q16 – Q17, relate to the source, type, direction, approximate size, of the contamination, vegetation, terrain, and seasonal restrictions. Provision was made for up to seven areas to be recorded in the report, supporting documents were the area, and village sketches. These reports proved 95% correct during the QA process.

Area information was assessed on 127 occasions, questioning by QA related to type of contamination, marking, verification of the village sketch and respondents. Of the reports checked, the most prevalent error related to omissions by the respondents.

To illustrate, in February 2001 an assessment in Banteay Meanchey Province identified that since the surveyors visit a week prior a mine had been located in an area within the village that had previously been assumed to be free from mines. The village leader requested that the surveyors revisit the village in order to amend the report to include the additional area. The report generated by the surveyors had been correct at the time of reporting but was classified as unsatisfactory because of this omission.

Socio Economic Information Q18 – Q20

Socio-economic information contained in the questionnaire Q18 – Q20, related to land use, ownership, impact, casualties, the supporting document was the Mine / UXO victims form.

Socio-economic information was assessed on 127 occasions, questioning by QA related to land ownership, casualties and respondents.

Of the reports checked, only casualty information required correction / additions on three occasions, once again this was done to correct omissions.

Respondents

Questions Q3, Q15, Q17, Q20 provide the details (name, sex, age, occupation) of the people that were information providers to the survey, typically respondents were easily located and it was possible on all but a handful of occasions to identify and locate them or at least a family member that could verify the conduct of the survey.

The surveyors had been directed to seek out respondents that had a long association with the village and were familiar with the area, this was done, and typically the respondents included landowners within the village that had some association with the management of the village. I.e. Village Leaders, Village Secretary, Members of the Village Development Committee, School teachers, Commune Leaders, and long term residents.

Spot UXO reports

95% of the reports assessed were satisfactory. Over 4,320 spot UXO reports were generated during the survey. Over 200 Spot UXO reports were assessed by Quality Assurance in the field.

On ten occasions the QA visit to the village resulted in additional UXO being reported, this did not reflect as a failure on the surveyors' behalf but more correctly as one of the limitations of the methodology.

Negative Village Survey Reports

99% reports were satisfactory. 7,306 negative village reports were processed by the database, 107 negative village survey reports were assessed by Quality Assurance in the field.

False Negatives Village Survey Reports

The two false negative villages were a result of villagers coming forward at the time of the QA visit and identifying that there were Spot UXO (2 items) in the village. In addition, over the survey period there were UXO accidents in villages identified in the survey as being negative for mine / UXU contamination. This does not indicate a failing by the surveyors but rather is a limitation of the methodology.

In addition, the revisit activity conducted in December 2001 identified that in areas in the North West of Cambodia, there are villages where Non Impact Cluster Bomb areas have not been captured by the survey. The question (Q 2:3 in the current survey form) was not asked, because it was not introduced into the questionnaire until non-impact cluster bomb areas were encountered later in the survey.

Non- Impact Cluster Bomb Areas

Non – Impact Cluster Bomb information was not collected in every village, the consequence of this is that there are villages that are very possibly false negative for Non – Impact Cluster Bomb areas.

False Positives Village Survey Reports

100% correct. A total of 326 positive village survey reports were assessed, on no occasion did the visit of the QA team result in the reclassification of a village from positive to negative.

Omissions

5 of the 486 reports assessed contained serious omissions that required the village to be revisited.

The greatest single factor that resulted in reports being rejected by Quality Assurance was the fact that the report contained omissions that had socio-economic consequences.

The fact that these omissions occurred reflects the limitations of the methodology employed, not a failure by the surveyors to perform their work

In every instance where omissions were identified the QA visit to the village resulted in additional information coming forward that was not made available to the survey detachment at the time of the survey. These were additional respondents, and previous respondents that had simply forgotten certain facts, or been made aware of additional information after the survey.

Database Queries on Quality Assured Reports

99.3% of the reports Quality Assured were accepted by the database. Of the 486 village survey reports Quality Assured then submitted to the Database three were later queried and returned to the field.

Those reports queried contained errors that did not relate to the information that was quality assured. I.e. The VPT coordinates were checked by QA and found to be correct, however QA did not make the enquiry, is this point (Village Leaders house) in the village?

Team Leaders Quality Control

91.4% satisfactory. The team leaders conducted quality control checks in 3,002 villages, of the villages quality controlled by the team leaders 116 (3.9%) villages were later assessed by QA, 10 reports were unsatisfactory.

The team leaders conducted control checks on more than 20% of the villages surveyed; they actually conducted more interviews than the surveyors, and averaged 600 interviews per team.

Given the other responsibilities allocated to the team leaders it may be that the 20% target was excessive. Team leaders Quality Control procedures were 100% verifiable and deemed to be adequate.

Field Managers Quality Control

The Field Managers conducted quality control checks in 262 villages, of the village's quality controlled by the field managers 9 (3.4%) villages were later assessed by Quality Assurance, 2 reports were unsatisfactory. The two occasions where the report were unsatisfactory were:

• The surveyors had failed to make the amendments directed by the Field Manager, and



• The village leader identified an additional area to the QA team, this after being visited by the Surveyors, then Team Leaders, then the Field Manager.

The Field managers Quality Control procedures were deemed by QA to be adequate.

Estimate of Most likely Error Rate in the Level One Database

The following table contains an estimate of the number of "discrete data entries" in the Database. The estimate is about 850,000 entries.

Database Statistics
Questionnaire Nos.	No. of Records	No. of Fields	Discrete Data Entities
1-4	13,908*	9	125,172
5-9	13,908	30	417,240
10-15**	1,640	35	57,400
16-17	2,947	35	103,145
18-20	2,947	32	94,304
Negative villages	7,486	7	52,402
*Two villages are recorded but have no information in the database	2	5	10
** If a village is negative seven (7) more fields of data are collected
Totals			849,653
Spot UXO	5,538	NA
Negative Village Reports	7,488	NA
Positive Village Reports	6,422	NA
Socio-economic	2,947

From the QA sample an extrapolation has been made in order to estimate the most probable error rate. This calculation is shown in the next table:

	Records	Sample size	% of Population	Reports with Errors	Likely Total   Errors	Likely Errors in Total Population	Assumed Errors per Report	Notes
1*	13,908	1,000	7.2%	11	22	306	2	All coordinates
2-9	13,908	486	3.5%	36	108	3,091	3	Not critical
10-15**	1,640	127	7.7%	0		0		No edits required
16-17	2,947	486	16.5%	5	50	303	10	Critical
18-20	2,947	127	4.3%	3	15	348	5
Negative villages	7,486	107	1.4%	2	4	280	2
Totals						4,328	0.51%	Error rate
Spot UXO	5,538	200	3.6%	10		277	5.00%	New reports
Negative Reports	7,488	107	1.4%	2		140	1.87%	False Negatives
Positive Reports	6,422	326	5.1%	0		0	0.00%	False Positives
Socio-economic	2,947	486	16.5%	5		30	1.03%	Omissions

As described previously each set of questions was Quality Assured at different levels of intensity depending on the type and significance of the data being collected.

It is with confidence a statement reflecting an error rate of less than 1% can be made. However, the L1S data is based upon the knowledge of local villagers, thus information unknown or incorrectly stated by village informants cannot, by definition be captured or corrected by the survey.

Summary and Conclusions

The close scrutiny and prompt remedial actions taken throughout the survey ensured a high quality of information gathered. The corrections made as a result of the QA/QC process have further contributed to the quality of this information.

The highest level of confidence is given to the integrity of the survey reports, and the compliance by the surveyors with standard operating procedures. Samples assessed by QA indicate not more than 1% of the reports contain errors or omissions with socio-economic consequences. The samples futher indicate indicates a possibly the survey contains a small percentage of villages that are false negatives for Spot UXO and Non-Impact Cluster Bomb Areas.

The survey was designed with three integrated procedural elements of Quality Control and Quality Assurance, this integrated approach has ensured a high quality survey process. Survey, Database, and Quality Assurance procedures integrated with open lines of communications between the three elements made for a high quality survey result.